Motivation

• Reduce etcd churn for high-frequency reconciliation and event data by using Redis as a fast, ephemeral telemetry store.
• Improve storage efficiency and clarity by separating workload spec and status persistence to allow independent updates and easier projection.
• Add observability for state-store operations to monitor where writes are occurring.

Description

• Added Redis dependency and configuration (REDIS_ADDR, REDIS_PASSWORD, REDIS_DB, REDIS_RECONCILE_TTL, REDIS_EVENT_TTL, REDIS_EVENT_MAX_ENTRIES) in internal/config and go.mod/go.sum updates to include github.com/redis/go-redis/v9 and other transitive modules.
• Implemented Redis initialization and telemetry helpers in internal/scheduler/redis_store.go, with fallback to etcd when Redis is unavailable, and functions to write reconciliation telemetry and event histories.
• Split workload persistence into workloads-spec/ and workloads-status/ prefixes by adding workload_projection.go, refactoring saveWorkload to persist spec and status separately, and updating keys/paths in state_store.go and other scheduler methods (e.g. GetWorkloads, GetWorkloadByID, DeleteWorkloadWithContext).
• Added a Prometheus counter state_store_writes_total and helper IncStateStoreWrite in internal/metrics and incremented it for spec, status, retry, assignment, reconciliation, and event write paths.
• Enhanced scheduler behavior: initialize Redis in NewScheduler, close Redis in Close, avoid unnecessary writes in UpdateWorkloadStatus, skip empty log writes in UpdateWorkloadLogs, and avoid persisting unchanged metadata in UpdateWorkloadMetadata.
• Reconciler now attempts to persist high-churn reconciliation metadata to Redis (with TTL) and falls back to etcd if Redis writes fail, and avoids redundant persistence for repeated NoAction results.

Testing

• No automated tests were executed as part of this change.

Codex Task